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ABSTRACT 

Problem-solving skills are often identified as a key component of 21 st century 
education. This study collected data from students enrolled in a university-level 
Liberal Education science course called Problems and Puzzles, which introduced 
students to the theory and practice of problem solving via puzzles. Based on class¬ 
room observation and other qualitative data collected over three semesters, we 
have identified three significant changes in student behaviour at specific points 
in the course. These changes can be posited to reveal three underlying threshold 
concepts in the evolution and establishment of students'problem-solving skills. 
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INTRODUCTION 

Problem solving is recognized among the critical attributes identified in Kay’s 21 st 
century learning framework (2010). Specific problem-solving skills such as analysis, trans¬ 
fer and metacognition also figure prominently in the revised Bloom’s taxonomy (Bloom, 
1956; Krathwohl, 2002) and in the facets of understanding described by Wiggins and Mc- 
Tighe (2005). But how might we ensure that our students learn and practice such skills? 

At the post-secondary level, problem-solving courses are regularly offered within the 
confined contexts of particular disciplines such as engineering, business or medicine, or to 
develop specific and distinct skills in subjects like mathematics or computer science. The 
problem-based learning approach (Savery, 2006; Wilkerson & Gijselaers, 1996), which 
grew out of education for medical students, also tends to focus on student learning in a 
content area through the use of open-ended problems in that area that students are re¬ 
quired to research and address. These approaches are generally disposed to using problem 
solving to teach content rather than using content to explicitly teach problem solving skills. 

We have studied the development of problem-solving skills in a less content-focused 
environment, as a learning outcome rather than using “problem-based learning” to achieve 
other content or context specific learning objectives. The locus of our study is a Liberal 
Education problem-solving course for university students called Problems and Puzzles. 
The course approaches the study of problem solving and the development of problem¬ 
solving skills in a general way, using puzzles as a learning vehicle to help students develop 
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a generic and transferable set of problem-solving skills. Pedagogically, we are interested 
in whether there are particular skills or strategies that are helpful to students at different 
times during the course, and in our research study we have looked for changes in student 
behaviour that could indicate development of strategic problem-solving skills. We also 
wish to explore whether observable behavioural changes are related to threshold con¬ 
cepts, in the sense suggested by Meyer and Land (2003). Meyer and Land define thresh¬ 
old concepts in a particular area of study as concepts which students must understand 
in a deep way in order to truly understand the area. Such threshold concepts share five 
key characteristics: they are irreversible, integrative, bounded, initially troublesome, and 
transformative. Once students attain a deep understanding of such a concept, there is no 
going back; the new understanding integrates all previous knowledge into a transformed 
understanding of the subject, and also delineates its boundaries from other related sub¬ 
jects. Such knowledge, and especially the process of gaining it and transforming one’s un¬ 
derstanding, can often be difficult and troublesome for students, as it involves changing 
and rearranging previous conceptions and misconceptions. 

The early literature on threshold concepts focuses on specific content-based concepts; 
for instance the main examples given in the seminal paper by Meyer and Land (2003) are 
opportunity cost in economics, complex numbers and limits in calculus, and signification 
in literary and cultural studies. More recent papers deal with a wide range of disciplines, 
including among many others history (Adler-Kassner, Majewski & Koshnick, 2012), 
engineering (Adawi & Kabo, 2012) and project management (Hokstand et.al., 2012). 

Schwartzman (2010) argues that, while Meyer and Land “construe approaches to 
teaching within the [threshold concept] framework as entirely discipline-specific, and have 
eschewed those that transcend disciplinary boundaries” (p. 22), there are crucial common¬ 
alities across disciplines in how students confront and deal with difficult or troublesome 
knowledge. Middendorf and Pace (2004) have described a “decoding the disciplines 
model [for] helping students learn disciplinary ways ofthinking” (p.l) which can be ap¬ 
plied broadly across all disciplines, and Diaz, Middendorf, Pace and Shopkow describe 
“bottlenecks” as “places where significant numbers of students are unable to grasp basic 
concepts or successfully complete important tasks” (2008, p. 212). We too wish to broaden 
the scope of threshold concepts. Our course and our research approach are both more 
process-oriented than content-oriented, and use a metacognitive lens to look at thresh¬ 
old processes as they might apply in the general development of problem-solving skills. 

THE COURSE 

The Problems and Puzzles course is taken as a science elective by a broad range of 
students from a number of faculties (Arts and Science, Fine Arts, Management, and Edu¬ 
cation) and a wide variety of majors. As a Liberal Education course, it introduces students 
to the problem-solving literature from a broad perspective, including history and anthro¬ 
pology of problems and problem solving, some mathematical problems and approaches, 
cognitive and metacognitive approaches, and connections to larger “real-world” problems. 
It presents a number of problem-solving and metacognitive strategies, and most impor¬ 
tantly, throughout the semester provides students with significant hands-on experience 
tackling and solving new problems in the form of puzzles. The class meets for a fifty- 
minute period three times a week, for a total of thirteen weeks. Although the strategies, 
topics and puzzles used are carefully sequenced, lecture instruction during class time is 
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kept to a minimum, with the focus on students working collaboratively or independently 
in class to apply the problem-solving skills and concepts taught to the solution of actual 
problems presented as puzzles. In almost every class students are given time to work on 
a new puzzle, followed by a class “debriefing” of different solutions and strategies used. 

THE STUDY 

The research described here is one component of a three-year on-going mixed- 
methods research project. To date we have collected a variety of quantitative and quali¬ 
tative data for each of the three times this course has been offered; from a total enrolment 
of 140 students, we had n = 100 study participants. We obtained research ethics approval 
from our institution for this on-going study (Protocol # 2011-084), and efforts were made 
to ensure appropriate confidentiality and anonymity of participants. Students were in¬ 
formed of the on-going research, were given the choice to participate, were able to opt 
out at any time, and were assured that the instructor of the course would not know who 
was participating. Data was collected, coded and analyzed by a research assistant, after 
the course had ended and all grades were submitted. 

The data we collected consists of three main types. First, we conducted in-class ob¬ 
servations, by the course instructor, another faculty colleague and a teaching assistant, 
the latter two of whom each attended about two-thirds of the classes. Our quantitative 
data consists of students’ responses to three Likert-scale surveys: a demographics and at¬ 
titudes survey conducted both pre- and post-course, and thinking style (Gregorc, 1979) 
and learning style (Barsch, 1991) surveys. Analysis and discussion of quantitative data has 
been reported more specifically elsewhere (Wismath, Orr & Good, 2014; Wismath, Orr 
& Zhong, 2013; Wismath & Zhong, 2014). Qualitative data collected from the voluntary 
student participants in this study forms the primary source for the component of our re¬ 
search described here, and includes a series of guided reflection assignments that were 
assigned as part of the course (and graded for completion only), and transcripts from two 
different post-course voluntary focus groups. The reflection assignments (collected over 
three semesters) formed part of the metacognitive emphasis of the course, encouraging 
students to reflect on their problem solving and their progress, but were also designed to 
provide qualitative information related to aspects we wished to investigate, such as col¬ 
laborative work, metacognitive activity, and transfer of skills. To ensure that participants 
could speak freely about their experiences, we also had research assistants not connected 
in any way with the course or project conduct two small-group post-course focus-group 
discussions, in 2012 and 2013, using similar questions to those in the reflections. 

In this paper we are reporting primarily on the qualitative data we collected, using 
both in-class observations and participants’ oral and written comments. We describe our 
observation of several changes in student behaviour displayed during the course, and 
theorize how the patterns we observed can be linked to the general theory of threshold 
concepts, as discussed above. Naturally, research of this nature-based primarily on ob¬ 
servations, reflections and surveys of students in a unique context-evokes examination 
of the “validity” and “reliability” of the data collection and analysis processes (Mayan, 
2009). Mayan posits that research which is necessarily contextual in nature is better de¬ 
fined by a “rigor” which demonstrates “why . . . the findings of a particular inquiry are 
worth paying attention to” (p. 100). Such rigor may be linked to the “credibility” of the 
descriptions and explanations presented by the research (Janesick, 1998), and to the 
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contextual relevance of the knowledge gained (Altheide & Johnson, 1998). The results 
reported and paradigms presented in this facet of our study are certainly contextually 
specific, but are additionally, we suggest, credible in a broader applicability. 

THE PERSISTENCE THRESHOLD 

The first shift in student behavior we have observed in the Problems and Puzzles 
course occurred noticeably after about three weeks of classes. Until that point, observed 
student behavior varied widely when students were presented with a new puzzle to work 
on. Many students seemed unable to begin; they appeared to sit and stare blankly at the 
board or handout, apparently unsure where to begin, and often waited for the instructor 
or teaching assistant to come by and provide help. Student comments referred to feeling 
“very anxious because I never know where to start”, and to getting “really anxious when 
I see puzzles and I don’t even know where to start.” Others seemed to have some initial 
ideas about what to do, but would appear to get stuck at some point and then wait for 
help. By the fourth week of class, however, this behavior visibly changed: all students 
now started working on each puzzle on their own, appearing to have some idea of how to 
begin, and even if observed to be experiencing some difficulty would consult each other 
or their notes for help and keep working. Students appeared to move from being scared, 
anxious and unable to start or sustain progress, to a point where they now seemed pre¬ 
pared to engage with each problem, and appeared to have strategies and techniques they 
were confident in using. In the final reflection assignment, almost all the students alluded 
to this change, as they wrote about having a tool-kit of strategies, and knowing where to 
start and what to do when they experienced difficulties. Comments from the reflection 
assignments support this apparent change in behaviour. 

Just having a list of problem-solving strategies has helped me immensely! I 
also feel I have developed the knowledge of what to do when I become stuck 
on a problem. 

Now I know that when I am stuck on a problem there are countless ways 
to approach it, to break it down, to simplify it or to look at it from a differ¬ 
ent perspective that may lead to me finding a solution faster, and more ele¬ 
gantly than I would have otherwise. 

My most valuable problem solving strategy I learned was to try a strategy, 
but if it's not going anywhere with the puzzle, try another one, and then an¬ 
other. This has helped me to solve many puzzles during class time. 

I can look at the problems in more ways than I used to. This makes starting 
a problem easier and makes it easier to get past “road blocks" that I come 
across because I have become more open minded about what techniques I 
am willing to use. 

These comments illustrate the participants’ realization that, while having a tool-kit of 
strategies is useful, mental habits of patience and persistence are important parts of prob¬ 
lem solving, especially at points where one experiences difficulties (Polya, 1945). The 
strategies students learned seemingly helped them to start towards and persist to the 
achievement of a solution; and seemingly increased their confidence in their abilities to 
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do so. More importantly, we posit that a mental habit of persistence and willingness to 
keep working when confounded was developed. Our first suggested threshold concept 
then is the importance of persistence for success in problem solving. Almost all the stu¬ 
dents began the course reporting limited confidence in their abilities to tackle problems 
(Wismath, Orr and Zhong, 2013), and expressed a belief that the key attributes involved 
in successful problem solving were intelligence and creativity. While these are important, 
it is in fact often persistence which leads to the most success: knowing what to do when 
you are stuck, and being willing to keep going rather than giving up (Polya, 1945). Gain¬ 
ing a deep understanding that most problems are solvable with sustained effort we suggest 
marks a fundamental and irreversible threshold in students’ development of problem¬ 
solving skills. The “reliability” of this assertion may be suggested by the significant in¬ 
crease in reported levels of acquired problem-solving skill and confidence in those skills 
among student participants (Wismath, Orr and Zhong, 2013). 

THE PROCESS OVER ANSWER THRESHOLD 

A second change we observed in student behaviour and comments began around 
the eighth week of the course, but was not as universally apparent as the first shift. In¬ 
deed, it occurred more gradually, and only about a quarter of the students showed direct 
indications of this change. The first indication of this second change occurred after the 
class worked on a particularly long and challenging puzzle, which not everyone success¬ 
fully completed during class time. One student remarked that she knew the answer to the 
problem (a particular number) but that that answer meant nothing to her; instead she 
indicated a strong desire to know why that answer was correct and what that particular 
number had to do with the pattern she and others had observed in the problem. At this 
point, this student and some others appeared to have moved beyond simply being satis¬ 
fied to determine a correct answer, and seemed to begin to focus more on deeper under¬ 
standing, and more specifically on the problem-solving process rather than on the answer. 
Students at this point additionally demonstrated an interest in finding multiple solutions 
to a problem, and looking for faster or better solutions, rather than simply being content 
to find “the answer” and stop working. Two comments, one from a reflection assignment 
and one from the focus group, elucidate this attitude. 

What is central to the knowledge learned in this class is not just "this is 
the answer to this puzzle," it is HOW one got to the answer. It is the pro¬ 
cess that is important; this includes the moment one is confronted with a 
problem and how they proceed. 

I guess I do like a lot of puzzles for fun because I'm nerdy like that, but I 
found I would like, get the puzzles before, but now I know why I'm getting 
them which is nice because before I would look at something and be like 
‘oh, okay I know this is the answer it would be the right answer but I didn't 
know why it was the right [answer]. 

Another participant in the focus group described a similar shift, based on her ex¬ 
perience at writing puzzle solutions for regular homework assignments. Here again we 
suggest there is a change in attitude, from “I got an answer and I’m done” to a search for 
deeper understanding and self-efficacy. 
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[W]ith the assignments, [...] its like 'I got this and that’s the answer, but 
that was it’. And I think as the course went on to do well on the assignment 
you had to be able to explain yourself. And then it became “of course I’m 
going to explain myself - because that is how you do a puzzle now.’ [...] 
that's the way it makes most sense, because you have to explain it to get the 
answer, at least for me. 

Another facet of this transition in orientation from answer to process is the realization 
that mistakes are not an indication of failure, but rather an integral part of the learning 
process. As one student wrote, “I now know that I can use a mistake as a learning expe¬ 
rience.” Another student comment from a focus group discussion also reports this shift: 

[Y]ou [...] learn from your mistakes, like if that didn’t work, don’t stop 
there, find out why or use that to help try again. Whereas before [■■■] I 
would totally be, like, 'I’m done!’ 

The second threshold concept we posit, therefore, is a metacognitive awareness that 
the process of problem solving is more important than getting an answer. Although not all 
the students in our study appeared to reach a deep understanding of this concept, some 
of them clearly demonstrated a realization that the process of thinking one’s way through 
complex ideas is more important than getting “the right answer.” 

THE CAREFUL MODELING THRESHOLD 

A final apparent change in student activity in class was observed to occur during the 
last three weeks of the semester. At this time, the instructor and other observers became 
aware of a marked increase in questions being asked by students as they started tackling 
a class puzzle. Earlier in the course, students had generally been observed to plunge di¬ 
rectly into their work, asking questions only near the end of the class and often apparently 
only if they got stuck or needed a hint. We observed by this point, however, that students 
started asking questions immediately after their first or second reading of the problem. 
These were usually clarification questions, such as: “Do you think it means this?” “Can 
we assume that...?” “Are we supposed to find this, or that?” 

We link this change to the distinction between “novice” and “expert” approaches 
to problem solving identified in the cognitive psychology literature (Pretz, Naples and 
Sternberg, 2003). Lesgold (1988) describes how experts spend more time at the start 
of a problem, determining a suitable representation and approach strategy, and less time 
actually working out the details later. Novices, on the other hand, tend to quickly choose 
a model and strategy and plunge into the problem, but often spend more time backtrack¬ 
ing later if their strategy does not work out or they have missed important details. Many 
of the participants in our study, while they did not necessarily become experts, did seem 
to move beyond novice status, in the sense of learning to invest careful consideration to 
a model at the start of a problem, using context and information given, in order to avoid 
wasting time later. A number of student comments on assignments and in the focus group 
reflect this shift. 

I have started to be more careful when starting a problem and looking at 
what I have and want to decide what strategy would work best. I would 
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normally jump right into a problem and start trying to figure it out without 
looking at the problem at a whole. 

I used to have a tendency to skim over instructions and questions (even on 
tests) so first andfioremost, I know that carefully reading the problem is 
fundamental. 

I have learned the importance of reading a question carefully. Knowing 
what the individual elements of a problem are is so essential to finding a so¬ 
lution. I used to quickly skim problems and dismiss relevant information. 

I think the most important thing I learned in the class was to really think 
about the problem before attempting to find solutions. When I take the 
time to step back and assess a situation or a problem I can make a good 
judgment on which process to use and it ensures I fully understand the 
problem. I find that this can make a big difference on finding the most effi¬ 
cient solution. 

One of the most important problem solving skills [...] is to really under¬ 
stand the problem. I had a bad habit of reading the question and jumping 
in too fast. This sometimes led to me getting the wrong answer since I didn't 
always have the right information. I have learned to take my time and 
make sure I know exactly what the question is asking. 

Our suggestion then is that a third threshold concept in problem-solving involves the 
realization of the importance of careful and complete modeling of a problem before one 
plunges into tackling it. Interestingly, this relates to the careful reading and understanding 
of a problem or puzzle which is the first step in all the many versions of problem-solving 
steps based on the classic four-step method by Polya (1945). This was discussed in our 
course in the initial lectures, and yet for most students a deep understanding of this con¬ 
cept was not reached until near the end of the course. 

THRESHOLD CONCEPT CHARACTERISTICS 

From a pedagogical perspective, we believe that the changes in student activity and 
understanding which we have observed can be consciously and intentionally facilitated 
for students. We have also suggested that our observations reveal three key threshold 
concepts underlying the development of problem-solving skills. All are metacognitive 
concepts, dealing with an understanding of the processes of problem solving rather than 
a specific content, reflecting the importance of metacognition in problem solving (Flavell, 
1979, 1987). It is worth noting that, although we have identified these concepts in an 
order based on our class-room observations, they do not necessarily occur in that order 
for individual students. The first change occurred early on and for all students; the third 
change occurred late in the course, for about three-quarters of the students; the second 
shift, from answer-only to process-focused, occurred more gradually over the duration 
of the course, and fewer than half the participants showed direct evidence of this change 
in approach. Further study on the inter-relationships of these concepts would be useful. 

Threshold concepts are irreversible. They integrate an area of study, establish bound¬ 
aries distinguishing that area from other areas, are initially troublesome, and are transfor- 
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mative for those who grasp them. We believe that the comments from our study partici¬ 
pants do confirm these defining characteristics of threshold concepts. The three concepts 
suggested here appear to be irreversible, as ways of looking at problem solving: students 
who have mastered these concepts do not move backwards. These deeper understandings 
help students integrate into a new framework the various problem-solving strategies and 
techniques they have learned. The acquisition of deeper understanding is additionally, to 
some extent, dependent on a troublesome process. Most students began the course ner¬ 
vous and afraid, and often appeared to struggle with the open-ended nature of the in-class 
time. Although the course sequence and puzzles were carefully structured, with very little 
formal lecturing being done in class, students were expected to fully engage with their 
own learning and to teach themselves how to problem-solve through intensive experi¬ 
ence. Students were observed to struggle at first with what to do or what was expected 
of them. One student for instance worried about whether he was supposed to hand in 
solutions to in-class problems or memorize them for tests. In fact, the overall process of 
the course put students continually in that ‘‘liminal state” described by Cousin (2006), 
in which they appeared to feel uncomfortable uncertainty during their continuous trans¬ 
formation from one way of thinking and knowing to another. 

The transformative nature of the concepts, as realized during the course experience, 
was revealed in a number of student comments. Many of the students commented that 
they had previously found problems intimidating, but “now just because we have done 
this class we see that puzzles are sometimes fun and they are not as intimidating as they 
seem and that they are solvable.” 

One student who spoke in the focus group about her math anxiety talked about now 
“being a problem solver”, showing the transformation she had undergone: 

I found this class,.. .for me being a problem solver... helped me take a 
step back and kind of view it differently.... I'm not a math major, I don't 
take math classes; this is the only one I have ever taken. And I get really 
anxious when I see puzzles (and I don't even) know where to start. Now 
I.. .feel like I can - actually found a couple that actually, I can enjoy. 

The importance of an increase in confidence was clearly illustrated in one participant’s 
comments at the end of the course: 

I honestly think that I have developed the confidence in my problem solv¬ 
ing abilities to be better able to explain strategies and processes. [...] It has 
taught me that if I put my mind to a task or complicated assignment and 
stick to it, I will eventually find the answer I am looking for. Trying this 
course that was completely outside my comfort zone only developed charac¬ 
teristics that established the self-belief that I have the ability to achieve any¬ 
thing I put my mind to. 

The transformative nature of the mastery of the threshold concepts was particularly 
evident in one focus group conversation. A student remarked that “I think this class should 
be mandatory [...] for first year students, imagine how valuable this would have been in 
your first year of university.” Another student agreed that in “first year it would have been 
very, very helpful.” A third student then replied that “I think if I would have taken this in 
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my first year, it would have been significantly.. . more life changing.” For these students 
the course learning did indeed transform their academic life. 

DISCUSSION AND CONCLUSIONS 

Our theories about metacognitive thresholds in problem solving are based on sus¬ 
tained observation by three instructors over three semesters, as well as students’ oral and 
written comments. An obvious next step for our research is to develop a more precise way 
to measure such shifts, perhaps through direct questions on assignments or other more 
specific interrogation of students. In general, we acknowledge the difficulty inherent in 
attempting to identify specific markers of progress in the development of problem-solving 
skills, but offer our results as a starting point for further study. We also note that we do 
not measure whether students have in fact developed as problem solvers, other than by 
their own report. Our quantitative survey data suggest that students experienced a signifi¬ 
cant shift in confidence and perceived ability (Wismath, Orr & Zhong, 2013; Wismath & 
Zhong, 2013), but this is not an objective measure of problem solving skill. 

While it is clear that our society needs educated citizens who are good problem 
solvers (Conference Board of Canada, 2013), it is not always clear how good generic 
problem solving skills can best be developed in our students. Our study suggests three 
distinct threshold concepts in the learning of problem-solving skills. For those teaching 
problem-solving skills, we suggest that mastery of these concepts can be emphasized with 
learned experiences structured to facilitate development. For example, the first threshold 
can be encouraged by providing a sequence of introductory puzzles at the right level to 
illustrate a number of different strategies, along with a supportive classroom environment 
that emphasizes effort and engagement rather than competition or grades. Modeling 
metacognitive discussion in class, and requiring metacognitive reflection by students on 
assignments, can promote an increasing awareness of problem solving as a journey and 
not a destination or answer only. Allowing students to figure out how to learn from their 
mistakes can also enhance an appreciation of the process of problem solving, rather than 
the process of getting a quick right answer. 

We have suggested here a broadening of the study of threshold concepts, away from 
purely content-driven concepts to more metacognitive concepts, which might be used 
to study how students develop an understanding of processes as well as content areas. 

This suggestion has been illustrated by our work on threshold processes in the de¬ 
velopment of problem solving skills. Good problem solving requires a constellation of 
skills, attitudes and mental approaches that, once developed, transforms the way people 
look at the world around them. In this sense, a course in problem solving can become “a 
portal, opening up a new and previously inaccessible way of thinking about something” 
(Meyer and Land, 2003, p. l). 
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